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ABSTRACT 



This paper summarizes Chapter XI, "An Overview and a 
Projection" of the book "Educational Evaluation and Decision*-Making- " 
The major assessments of this chapter: (1) how good were the answers 

provided to older questions; (2) what new questions were now able to 
be raised; and (3) who might be in a position to deal with them, are 
discussed- Concerning the first point, the author sees the book as 
more than passably adequate in terms of meeting its objectives, 
meeting the criteria for a good evaluation^, responding to the five 
basic problems of evaluation, and meeting the conditions for an 
adequate theory. Regarding new questions, the areas of 
decision- maki ng, criteria, methodology of the interface role, 
methodology of the technical role, data systems, implementation and 
organization, training, and therapy, are seen as those most in need 
of inquiry and development. Finally, in relation to who might deal 
with these questions, the author includes an eight page quotation 
from the book manuscript to discuss this topic. It suggests that 
evaluation practitioners on the local, state and national levels, 
evaluation consultants, evaluation research and development 
personnel, users of evaluation reports, consumers of evaluation 
reports, related professional groups and funding agencies might 
contribute solutions to new questions, (AG) 
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The book which we have been sunnnarizing here today is obviously 
a beginning and not an end. The authors regard it as the first 
step in a continuing inquiry into evaluation^ and we are sure that 
you will agree with that judgment should you dip into the book your- 
self. I have found it useful to think of knowledge as forming a 
sphere, with all of the unknown forming the ” space” around the 
sphere. As knowledge grows, the sphere enlarges, but as it en- 
larges, it comes into contact with more and more of the unknowTi. 

So it is with this book. As a result of the inquiry we have con- 
ducted we have answered a few questions that we were able to raise 
given the state of our thinking three years ago« But we have 
raised more questions than we have answered as our knowledge en- 
larged. 

The last chapter of our book, which it is my task to recapit- 
ulate today, tried to make two major assessments: (1) how good 

were the answers that we provided to our older questions, and (2) 
what new questions were now able to be raised and who might be in 
a favorable position to deal with them? I shall also deal with 
each of these assessments in my summary. 

How Good are Our Answers to the Older Questions ? 

We suggest in the book that there are at least four ways to 
deal with this question: 

1 . How well t z ere the objectives that were set for the book 
actually met ? These objectives were the follov/ing: 

a. To expose five problem areas of evaluation: defini- 

tion, decision-making, values and criteria, levels, and the 
research model. 
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b* To identify and assess extant or emergent formulations 
that might be used in conceptualizing solutions to these five 
problems, and to fashion them into tentative solutions. 

c. To synthesize a new definition and methodology of 
evaluation that builds upon the products of (b) . 

d. To provide some operational guidelines for implement- 
ing and proposed new approach in terms of personnel, organiz- 
ation, and administration. 

Our conclusion : the objectives were met in the sense that they 

were all dealt with, but this leaves open the question of how well 
they were dealt with. No doubt the reader will have reservations on 
this point, as do the authors. But regardless of these reservations, 
the book asserts that *’the only real question ... is whether the 
objectives have been met well enough to warrant attempts to imple- 
ment the proposed model. If the model has sufficient face validity 
to meet this challenge, the reservations will either disappear, or 
be confirmed, as empirical data and experience mount.” The authors 
conclude finally on this point that ”they have made a sufficient 
case to warrant this next step,” i.e., implementation efforts. 

2. Meeting the criteria for an evaluation . The book outlines 
eleven criteria which a good evaluation must be able to meet, and 
so it seems proper to ask whether an evaluation designed on the 
principles outlined in the book can be expected to meet these cri- 
teria any more satisfactorily than could an evaluation designed on 
more ”classic” principles. 

On the scientific criteria of internal validity, external 
validity, reliability, and objectivity, there is scsie good news and 
some bad. Inten\al validity, it is asserted, may well be enhanced 
because of the exlstevice of context data that provide an excellent 
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baseline measure, because the emphasis on invited interference elimi- 
nates laboratory antisepsis, and because process data are available 
to investigate instances in which internal validity might be suspect. 
External validity is considerably enhanced, it is suggested, both 
because real world situations are emphasized and because process data 
provide insights about the conditions under which the findings are 

f 

applicable. On the other hand, reliability may suffer because cer- 
tain relatively unreliable data sources are still considered admiss- 
able (e.g., testimony and other self-reports), although it should 
be noted that the use of many different data sources provides cross- 
check possibilities that are often absent under more controlled con- 
ditions. Similarly objectivity may suffer because of the wide range 
of information admissable, but cross-check possibilities also exist. 
The authors conclude about the scientific criteria that *^on balance, 
the proposed evaluative approach does not suffer by comparison and 
in some particulars offers advantages of considerable significance.” 
It is on the practical criteria that the proposed approach 
really shines, however, and this is not surprising, since in a sense 
it was designed specifically with these criteria in mind. Relevance, 
importance, scope, and credibility are all assured, the authors as- 
sert, because of the interface relationship between the evaluator 
and the decision maker. Timeliness is well served because of the 
proactive posture of the CIPP model, as assured by the delineating 
steps. Pervasiveness is likewise assured by the interface provi- 
sions. The authors conclude, ”When assessed in terms of its capa- 
bility for providiiig evaluations that measure up well on the six 
practical criteria, one must conclude that the proposed approach 
has a great deal warrant it.” 
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A final prudential criterion of efficiency is also invoked on 
which the proposed approach, the authors warn, might well suffer 
unless great care is exercised. A full-scale system of the sort 
called for by the CIPP model cannot be justified on the basis of a 
few decisions to be served. The cost of servicing the decision 
might well exceed the cost of implementing it. The authors suggest 
that extreme caution should be employed until more empirical data 
are in concerning the real costs involved. 

3. Responding to the five basic problems of the book . The 
book suggests that there are five major problems with current evalu- 
ation, and defines these as the problem of definition , the problem 
^Gci sion-making , the problem of criteria and values , the problem 
of levels , and the problem of the research model . The question is 
raised whether the new formulations have any utility ’’in ameliorat- 
ing or overcoming these problems.” 

Generally speaking these problems are seen as reasonably well 
handled. A new definition is proposed which avoids many of the 
problems of older definitions, which enlarges the concept of evalu- 
ation, and assigns it a more central role in organizational processes 
The problem of decision-making is perhaps best handled because the 
new formulation is precisely designed- to relate evaluation to 
decision-making. The problem of criteria and values is addressed 
directly but while important steps are made, the theory is by no 
means fully or finally developed on this point. The problem of 
levels is at least well illuminated in that the decisions made at 
different levels are seen to require different information and hence 
different (not merely aggregative) evaluations, finally, the preb- 
O lem of the research model is least well handled in that, while the 
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problem is well delineated, very fev; solutions are proposed. 

4. Meeting the conditions for an adequate theory . Since what 
is proposed in the book is, in a sense, a new evaluation theory, it 
is useful to ask to what extent the formulations meet general re- 
quirements for theoretical adequacy. This difficult philosophical 
question is' by no means systematically handled but a minimal attempt 
is made to show that the formulations of the book are coherent , 
i.e., make it possj.ble to reformulate or account for other theoret- 
ical statements about evaluation; that they are internally consis - 
tent ; that they are comprehensive , in the sense of forming a com- 
plete system; that they are reasonably well able to generate test - 
able hypotheses ; that they can gene^rate good operational designs ; 
and that they provide a general guide for a wide range of educa- 
tional evaluation problems. On balance the proposed formulations 
meet general theoretical criteria reasonably well. 

When viewed in terms of these four ways of assessing the ade- 
quacy of the proposed formulations, i.e., meeting the book’s objec- 
tives, meeting the criteria for a good evaluation, responding to the 
five basic problems, and meeting the conditions for an adequate 
theoiry, the book is seen as more than passably adequate, although 
certainly far from perfect. The answers to the older questions, in 
short, are not too bad. 

What are Some of the New Questions Raised by the Book ? 

X quote from the book: 

This book is a culminating experience for its seven 
authors because it Is the end product or years of indivi- 
dual thinking about evaluation and of many months of group 
debate, reassessment, and ‘compromise to accominodate our 
several points of view. We believe that the product is 
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as good as x^c caa make it at this time. But precisely be- 
cause we have thought about it so much, and haggled so 
long over its every detail, no one is more aware, or more 
acutely appreciative, of its shortcomings than we. It 
seems appropriate to us, following our o\m tenets, that 
we should conduct a kind of context evaluation, concerned 
with assessing the present status of the emergent theory 
and with identifying the needs, problems, and opportunities 
that still confront us, and others who may wish to join 
in • . • • 

The last chapter then goes on to describe eight areas of need, 
problems, or opportunities: (1) decision-making, (2) criteria, 

(3) the methodology of the interface role, (4) the. methodology of 
the technical role, (5) data systems, (6) implementation and or- 
ganization, (7) training, and (8) therapy. These are the areas 
which the authors assert are most in need of inquiry and develop- 
ment if the theory proposed here is to be pushed ahead in produc- 
tive fashion. 

1. Decision-making . Two major problems are seen here. The 
first deals with the adequacy of the theoretical formulations pro- 
posed in the book. The authors write that ”it has been a sourca of 
great concern . . . that they have not been able to identify an 
existing theory of decision-making that has permitted a heuristic 
application to the evaluation problem. The writing on decision- 
making . . . has been *raanufacturcd* by the authors,** r^.ther than 
adapted frora existing formulations. This is a most serious defici- 
ency since,* as the authors state, their **own lack of expertise in 
the area gives them pause about so cavalier an action.’* 

A number of formulations were devised by the authors to fill 
this gap. These include concepts about the decision process 
(awareness, design, choice, and actio.a) ; about decision setting s 
(me t amor p hie, hoineo static, inc r erne n t a 1 , and ne omo b i 1 i s 1 1 c ) ; about 
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decision models (disjointed incremental, synoptic, planned change, 
and others); and about decision types (planning, structuring, im- 
plementing, and recycling). Each of these formulations rests upon 
an ad hoc rationale generated in the OTiting, but these rationales 
may or may not be valid when analyzed from other conceptual or em- 
pirical po5-nfs of view. Both conceptual and empirical critiques are 
urgently required. 

The second problem in the area of decision-making has to do 
with the decision maker himself. As the book observes, the ’*lack 
of empirical information about real-world decision-makers is appall- 
ing.” Most at issue is the question of whether decision-makers in 
fact act rationally. Second, there is the question of whether even 
rational decision-making must not take account of a whole series 
of non-rational elements, e.g., the political realities. The cir- 
cumstances, the authors point cut, "argiie against a purely rational 
decision-making model .... Obviously a better model, based on 
empirical infomiation about real-world decision makers, is very 
much needed.” 

2. Criteria . Criteria are necessarily central to any formul- 
ation in which decision-making is such an important element, since 
decisions ”are not made in a vacuum but always in relation to some 
explicit or implicit value structure-." Four areas of difficulty 
are noted. First, the formulations of the bock are again dratm up 
largely out of context to the literature on values, partly because 
the authors can claim little expertise in this area, but also 
because the "literature does not deal with value problems in a v/ay 
that makes heuristic applications to the evaluation problem easy.” 
Again close conceptual and empirical critiques are needed. 
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Second, the identification of criteria is seen as a lengthy and 
continuing interface problem. Decision-makers* values arc not In 
every instance overt and ;^ell-appreciated. The decision-maker, like 
all humans, has difficulty in identifying his values; many remain 
hidden because of his lack of insight. Others will not be revealed 
because they constitute the decision-maker's ^’hidden agenda." Hence 
the evaluator faces a real dilemma: provide information on these 

little knoTvn or hidden values and risk censure or rejection, or 
fail to provide information and hence not service decisions ade- 
quately* 

A third values problem lies in the fact that often more than 
one decision-maker (or at least, more than one legitimizing group) 
may be involved, so that values may be inconsistent, discontinuous, 
or even in conflict. ”The question of which value structure to at- 
tend to and how to resolve the conflicts, inconsistencies, or dis- 
continuities which exist must be dealt with." 

Finally, "the evaluator must realize that value structures are 
dynamic--they change with time and circumstances .... This 
dynamic process itself requires study and theoretical explication." 
3. The methodology of the interface role . The authors assert: 

A most glaring and conspicuous omission in this book is 
the failure to provide operational guidance for the evaluator 
as he plays ^thje/ interface role. Obviously the playing of 
it calls for a very sensitive interpersonal relationship. 

We have made the point in several places that it is this in- 
terface relationship that provides the greatest sanction for 
labelling the evaluator as a "professional," in the sense of 
a close, even pri^/ileged relationship with a client. If 
this relationship is so important why is it not dealt with in 
greater detail here? Jvist what does an evaluator ^ when he 
plays this role? 

The answer to that question at this point in time is that 
mainly, we do not know. The mode of creating awareness, of spelling 
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out decision alternatives, of working out criteria, of **educating’* 
a decision-maker, of reporting information productively, and of 
dealing with the inevitable questions of ethics and objectivity are 
largely unknown. Both conceptual and empirical explications are 
urgently required. 

4. T he methodology of the technical role . Devising a methodo- 
logy appropriate to the technical roles of collecting, organizing, 
and analyzing data may, the authors assert, turn out to be even 
more difficult than devising a methodology for the interface role, 
for the latter is still an open question while it is often assumed 
that the former methodology is well understood and eminently teach- 
able. But this is far from the true state of affairs, as much of 
the writing of the book seems to show. A whole new methodological 
approach based on assumptions other than those that undergird much 
of the conventional research methodology is needed. Many of these 
assumptions are inappropriate vjhen the interest of the inquirer is 
in answering evaluative rather than research questions. Others 
are no longer necessary because of advances in technology, chiefly 
the computer. *‘The construction of a nev^ general theory for col- 
lecting, organizing, and analyzing data will be a formidable task,” 
the authors assert. ”The analysis / o f the book/ seems to indicate 
that it is possible but provides little insight into how it should 
be done.” 

Moreover, other technical advances will be required to augment 
the new theory when it is developed. New instrumentation, particu- 
larly instrumentation capable of handling classically ’’intangible” 
variables, will be needed. New theories of evaluative design will 
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be requ?lred* Unobtrusive measures will be needed, particularly in 
the area of process evaluation. 

The authors conclude, '’The task of devising suitable methodolo- 
gies for the technical role will not be a simple one. .The most com- 
petent statisticians, systems analysts, computer specialists, and 
the like will be required, and the challenge will be deserving of 
their best efforts." 

5. Data systems . "The kind of evaluation that has been pro- 
posed here mandates the existence of data systems capable of hand- 
ling the masses of information that will be required. These data 
systems will have to be capable of storing and retrieving not only 
large amounts of information but a number of varieties of inform- 
ation as well." 

A context monitoring system may require including hundreds of 
different variables, each of which must be relatable to all others 
on .2 random basis* Not only will individual variables have to be 
stored and retrieved, but systems of variables as well, as for 
example, in the case of the alternatives that exist in response to 
some need, problem, or opportunity that a context evaluation has 
exposed. 

Non- numerical data must also be available, in some cases in 
highly complex form. So for example, an input evaluator seeking 
to devise a new curriculum may want to know about the impact of in- 
formation on motivation on his work. He wants this information not 
in the form of a bibliography of articles about motivation research 
but in the* form of a set of pre-interpreto d principles that he can 





directly apply 
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Data systems capable of such feats of storage and retrieval 
will not be inexpensive, and this fact raises questions of economic 
feasibility. Probably consortia of agencies will need to be formed 
to place such a data system at the disposal of all educational agen- 
cies that raight require it. How can such networks be formed and 
operated to produce maximum effectiveness and efficiency? 

The authors conclude their discussion of data systems with the 
following observation: 

Unless these questions can be satisfactorily answered 
within the bounds of resources that it is sensible to commit 
to these purposes, the kind of evaluation system projected 
in this book is simply not feasible. This does not mean 
that no evaluation can take place until such facilities are 
available; it does mean that if the evaluation system is to 
have utility and the payoff contemplated it must ultimately 
rest upon an adequate data system. 

6. Implementation and organization . The book does contain 
several case studies of existing organizations that are making an 
effort to operationalise the model of the book. But, the authors 
note. 



• • • the reader has no doubt already discovered the 
acute shortcomings of these formulations; anyone who searches 
these chapters for help in setting up his own evaluation 
unit will find only a few rules of thumb, some incomplete 
ease descriptions that illustrate more the fact that units 
can be established rather than how to go about doing it, and 
a few exhortations. 

A number of steps are called for by the authors to provide the 
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kind of practical help that is needed. These include: carrying out 

a wide variety of more scientifically based case studies to provide 
a sound empirical basis for organizational decisions; devising gener- 
al strategies that take account of the differing circumstances that 
inhabit different decisions settings (e.g., incremental strategies 
for situations in which e gc-slow approach is counseled and 
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nGomobilistic strategies where an all-out effort is possible); and 
devising appropriate tactics in support of each type of general 
strategy, e,g. , tactics appropriate to staffing, budgeting, staff 
morale, and similar administrative questions. 

7. Training . The authors observe that ”it may well be the case 
that the lack of trained personnel to carry cut evaluations ... is 
the single biggest stumbling block that confronts the profession .... 
The problem has both quantitative and qualitative aspects. Very few 
trained evaluators exist and few are being trained by present means. 
Moreover, those who have been trained or are being trained are, in 
the main, * traditional* evaluators . . . .” 

There is little reason to hope that this problem will soon be 
ameliorated, for there are simply no programs that will do an adequate 
job of training. Perhaps the best evidence for this statement is the 
fact that those persons functioning in the field as evaluators tend not to 
turn to universities for training but are content *^to train themselves 
through private reading, attendance at 'occasional and short term 
evaluation workshops, etc. Universities are certainly not inundated 
with large numbers of applicants. ... 

Why do these programs not exist? The authors suggest several answers 
and thereby imply the range of development work that is necessary if 
this problem is to be eliminated. There is first of all almost no 
agreement on what is an adequate role definition for an evaluator. Both 
adequate conceptual formulations and good empirical data about actual 
role expectations are lacking. There is, further, almost complete lack 
of experience with the interface and the administrative roles of the 
evaluator, as defined in the book. Professorial staff competent to 
s'an training programs are themselves in critically short supply^ Traiuxng 
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materials, even in the sense of conventional text books, are lacking. 

Finally, there is a lamentable lack of resources, particularly federal 
resources, for attacking the problem. 

Therapy . The term therapy" is used to coincide with a recurrent 
metaphor used in the book, which is that of evaluation’s "illness." 

Indeed, the book begins with a listing of certain "symptoms" of that 
illness, and it is to the amelioration of these symptoms that therapy is 
required. The authors assert that "symptoms disappear when illnesses 
are cured, and we may confidently assume that if better evaluations 
become possible, that /Jhe symptoms of evaluation’s illnes^/ will also 
disappear. But that will take time. ... Is there nothing that can be 
done to provide at least some symptomatic relief in the interim? Can we 
not identify at least the equj.valent of an antihistamine that will dry 
up our noses even if not curing our colds?" 

The authors are able to make a few suggestions. The first is that 
we generally admit that evaluation is in fact "ill," to "clear the air and 
rid us of the need for foolish posturing." Second, we need to devise 
a strategy to reach the consultants who work with practitioners on evaluation 
problems, and who must therefore bear much of the responsibility for the 
current state of affairs. Third, we "need to make it more . . . rewarding 
to do reasonable evaluation." Fourth, we should "make it easier for 
local agencies that wish to engage in evaluations to do so, for example, 
by setting up evaluation service bureaus to render assistance. Finally, 
we must invest funds in long range planning that "will serve as a visible 
sign of corrimitment that we. intend to do something about the evaluate. on 
problems that confront us. . . . Nothing comforts the patient more than 
some visible effort on the part of his physician ... to cure him, even 
if he knows the cure will take a long time." 



-14- 



All of these efforts at therapy require design and development, 
as well as support, and we must turn some effort in this direction, the 
authors seem to assert. 

These eight areas, decision making, criteria, the methodology of the 
interface role, the methodology of the technical role, data systems, 
.implementation and organization, training, and therapy, provide the most 
immediate and pervasive challenges to be overcome if positive next steps 
are to be taken. 

Who Can Effectively Participate in Talcing These Next Steps? 

I believe that this question is important enough to warrant my 
quoting the relevant sections of the book in their entirety. The 
remaining portion of this paper will thus be a full length quotation of 
pp. 523-530 of the mimeographed manuscript. 

Who should Participate in Resolving the Unresolved Issues ? 

There is certainly no existing group of professions whose training 
encompasses the many role behaviors that have been ascribed to evaluators 
in this book. Indeed, there is a great deal of question whether it is 
either possible or desirable to train a single person in all of these 
competencies; it might perhaps be more sensible to speak of a set of 
interlocking evaluator roles (e.g., interface and technical roles, to 
give an obvious instance) than a single evaluator role. Even if such a 
set is contemplated, however, the range of activities required to respovid 
to all of the issues raised above, and to others which the reader may 
have detected, requires the cooperation and insight of a wider range of 
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professionals than could ever be subsumed under the simple evaluator 
rubric. The listing ^-/hich follows despite its length is illustrative 
rather than exhaustive of this wider range. 

We may distinguish at once between primary and secondary groupo. 
Primary groups consist of individuals who are directly engaged either in 
,the conceptualization or practice of evaluation. Secondary groups are 
those who use the results of evaluations or who can contribute ideas and 
tools that might be used by primary evaluators. Within each of these two 
groups a number of subgroups may be distinguished. The listing below 
provides some brief definitions and examples; 

1. Primary groups 

A. Evaluation practitioners . These are the persons who 
actually design and carry out evaluations, and may 
range from highly technically oriented personnel such 
as instrument developers or computer prograinrners 
through highly client oriented personnel characterized 
here as "interface*' personnel. Their major contri- 
bution typically is action . It is useful to dis- 
tinguish further in terms of at least three levels 
(chiefly because of levels problem considerations) : 

practitioners such as are found in local 
school districts, Title III projects, Head Start 
activities, and the like; 

State or regional practitioners , such as those in 
state departments of education, regional educa- 
tional laboratories, interstate projects, and the 
like; 

3. National practitioners , such as those in the United 
States Office of Education, Project Talent, 

, national assessment, and the like; 

B. Evaluation consultants . These are persons called in, 
either on an ad hoc or continuing basis, to assist in 
carrying out evaluations, to critique them (acting as 
evaluation "auditors"), or sometimes to take opera- 
tional responsibility for seme segment of the evalua- 
tion. Their major contribution typically is advice or 
help . It is again useful to distinguish three types 
on the basis of their parent organizations: 
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1. Practitioner organization based consultants , that is, 
consultants who themselves come from an organization 
much like the one they are advising, i.e.r another 
local, state, regional, or national unit* These 
consultants are utilized mainly on the basis of their 
experience. 

2. University based consultants . These consultants are 
typically utilized because of their technical 
expertise . 

3. Private organization based consultants , often dra^^m 
from either profit or non-profit agencies that are 
in business to render service. Such consultants 
may have either experience or expertise (and often 
have both) but their chief utility seems to be their 
availability and their willingness to deal with the 
client* s problems on the client* s terms . 

C. Evaluation research and devel op ment personnel . These are 

the persons are engaged directly in the development of 
theories, models, paradigms, and the like intended to pro- 
vide the conceptual basis for evaluation and/or the persons 
engaged in the development of instruments, designs, data 
processing techniques, and the like intended to provide 
the operational basis for evaluation. Their major contri- 
butions are thus ideas or tool s. Again it is convenient 
to project three types: 

Those based in education departments or schools in 
universities . Robert Stake (University of Illinois) 
and Marvin Alkin (UCLA) are examples. 

2. T hose based in universities but not in education units . 
Michael Scriven (University cf California) and the late 
Edward Suchman (University of Pittsburgh) are examples. 

3 * Those based outside universitie s, often in practitioner 
organizations or private consultant or research firms. 
John Flanagan (AIR) and Wayne Wright stone (New York 
City Schools) are examples. 

11. Secondary groups 

Users of evaluation reports . In the language of this 
book these are the decision-makers whose decisions are 
being serviced by the evaluation activities. 

Consumers of evaluation reports . These are typically the 
publics or patrons of the organization or agency for whose 
decision-makers the evaluations wore performed, and may 
include other members of the organization who are not: de- 
cision-makers but are affected by the decisions made. In 
the former category we may mention such examples as parents, 
taxpayers, or special interest groups such as the ^raerican 
Legion. Exair.ples of che latter category are teachers, 
pupils, or subcontractors and suppliers. 

1 
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C. Related professional groups * Included in this category 
are a variety of relatively more basic sciences on which 
the relatively more applied science of evaluation may 
draw. Included are a£ least the following: 

1. Measurement scientists and, instrument development 
specialists 

2. Statisticians # 

3. Computer science specialists 

4. Systems analysts and operations re searchers 

5. Philosophers , especially value philosophers 

6. Theorists in organization and administration 

7. Communication theorists 

D. Funding agencies . These are the foundations or govern- 
mental agencies that can support necessary research and 
development effort or can underwrite the extraordinary 
installation costs of evaluation systeras. 

What Can Each of These Groups Contribute ? 

It is obvious that each of these groups can make some particular 
contribution. Some groups have a greater stake in the matter and hence 
might be expected to contribute more; others may not see themselves as 
involved directly with evaluation but may nevertheless contribute a key 
Idea, tool, or action. Each group brings a particular perspective and 
also a particular set of interests and competencies. What is probably 
most needed is a symbiotic relationship among the groups that will result 
in the potential contribution of each being utilized at the appropriate 
time and in the appropriate way. The major responsibility for effecting 
this symbiotic relationship clearly must be placed on the primary groups, 
and among them, on the research and development personnel who, on balance, 
would seem to have the greatest amount of time, the most institutional 
freedcra, and the necessary intellectual scope to get the Job done. 
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The question of what contribution should be Tiiade by each of the 
groups listed in the preceding section is probably best left to the dis- 
cretion of each individual group; it would be presumptuous to attempt to 
guage the degree of commitment to the problem ox the amount of insight 
that might be brought to bear by each. Some exemplary contributions are 
outlined in the accompanying Figure 1, however, to provide the flavor of 
what might be done. This figure indicates the various contributing groups 
and relates them to the eight areas of unresolved issues dealt with in 
Section 3 of this chapter. Example contributions are indicated by checks 
in the appropriate cells. It should be recalled that we are concerned 
here with the question of resolving issues or questions, not in engaging 
in significant activity in the cells in question. 

So for example, local practitioners may very well be engaged in opera- 
tional activities concerned with each of the eight cells; however, it seems 
unrealistic to expect them to contribute much toward resolving the question 
of an appropriate decision-making theoiry to service evaluation, or an 
appropriate interface methodology. Local practitioners might make signif- 
icant contributions to the organization and administration dilemma, however, 
for they must devise some kind of organization. Experience with a variety 
of local units might give some insights into the most efficient or viable 
organizational patterns. This problem would be especially aided if local 
administrators v;ould keep detailed case histories of the development of 
their units. Some local practitioners might similarly contribute to data 
systems; this cell has not been checked, however, on the assumption that 
most local evaluation units would not set up their own data storage and 
retrieval system but would instead engage in sharing with other units. 
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SOME EXEMPLARY COMTR I BUT ! 0!!S TMAT HiGHT 5E ^ADE 
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Both state snd national level practitioners might, on similar 

\ 

grounds, make significant contributions to data systems and organization. 
Accordingly these cells have been checked. 

Consultants at all levels seem to be in a particularly useful posture 
In relation to the last four issue areas which seem to be more practice 
oriented than the first four. Thus they seem to be well able to provide 
advice to practitioners about data systems and organizational problems, 
and in turn to reflect problems encountered at the level of practice to 
the research and development personnel with whom they interact. Consultants 
are also able to provide training, particularly in the form of workshops 
or direct on-the-job instruction. Finally they seem to be in a very 
fortunate position with regard to providing therapy for the practitioners 
whom they contact. 

Research and development personnel as noted above bear the primary 
responsibility for tying the whole system together. It is up to them to 
meld available ideas into coherent systems and to provide the basis 
the tools, if not the tools themselves, that practitioners will use. All 
three kinds of R & D personnel displayed in Figure 1 can make contributions 
to interface and technical methodologies. The non-education R 6: D 
personnel are probably best able to deal with the decision-making and 
value areas. University personnel, in or out of education, can make 
primary contributions to training, including especially the development 
of training experiences and materials; they are probably also best 
equipped to deal with the organization-administration avea^ Finally 
certain kinds of non-university R & D personnel, e.g., those employed by 
data equipment manufacturers, can probably best deal with the data 
systems area. 
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Among the secondary groups, users and consumers can make a real con- 
tribution not so much in direct work on issue resolution but by making 
themselves available to R & D personnel in the areas of decision-making 
and value. These are the groups that make decisions (primary or secondary) 
and who implicitly or explicitly hold the values that become applied in 
.the form of criteria. By making themselves available for study and by 
cooperating in exposing their own processes they can help enormously. 

Each of the related professional groups relates well to one or two of 
the issue areas and can contribute ideas, insights, or tools that would 
help in their resolution. Measurement specialists and instruments builders 
are necessary in devising technical methodologies, as are statisticians. 
Computer specialists are also needed in this area but primarily, of course, 
in the data systems area. Systems analysts have much to say about decision- 
making, criteria, and particularly about interface methodology. Philoso- 
phers are needed to devise better approaches to the value area and to 
interface methodology. Organizational theorists illuminate decision- 
making, interface methodology, and of course organization and admlnis- 
trat ion. Communication theorists can contribute both to interface and 
technical methodology and to the theory of data systems. 

Finally, funding agencies have a vital role to play in every area, 
ill underwriting research and development costs, in supporting agencies in 
institutionalization of the new evaluation, and in general lending their 
good name in the cause of better evaluation. 
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